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We study the performance of different message passing algorithms in the two dimensional 
Edwards Anderson model. We show that the standard Belief Propagation (BP) algorithm 
converges only at high temperature to a paramagnetic solution. Then, we test a Generalized 
Belief Propagation (GBP) algorithm, derived from a Cluster Variational Method (CVM) at 
the plaquette level. We compare its performance with BP and with other algorithms derived 
under the same approximation: Double Loop (DL) and a two-ways message passing algo- 
rithm (HAK). The plaquette- CVM approximation improves BP in at least three ways: the 
quality of the paramagnetic solution at high temperatures, a better estimate (lower) for the 
critical temperature, and the fact that the GBP message passing algorithm converges also to 
non paramagnetic solutions. The lack of convergence of the standard GBP message passing 
algorithm at low temperatures seems to be related to the implementation details and not to 
the appearance of long range order. In fact, we prove that a gauge invariance of the con- 
strained CVM free energy can be exploited to derive a new message passing algorithm which 
converges at even lower temperatures. In all its region of convergence this new algorithm is 
faster than HAK and DL by some orders of magnitude. 
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I. INTRODUCTION 



The 2D Edwards-Anderson (EA) model in statistical mechanics is defined by a set a = 
{si . . .sn} of N Ising spins Si = ±1 placed on the nodes of a 2D square lattice, and random 
interactions Jj j at the edges, with a Hamiltonian 

= - } y JijSiSj 

<i,j> 

where < i,j > runs over all couples of neighboring spins (first neighbors on the lattice). The 
Jij are the magnetic interchange constants between spins and are supposed fixed for any given 
instance of the system, and the spins Si are the dynamic variables. We will focus on one of the 
most common disorder types, the bimodal interactions J = ±1 with equal probabilities. 

The statistical mechanics of the EA model, at a temperature T = is given by the Gibbs- 
Boltzmann distribution 

P(a) = - where Z = ^ e -^W 



The direct computation of the partition function Z, or any marginal probability distribution like 
p(si,Sj) = J2a\s z sj P( a )i 1S a time consuming task, unattainable in general, and therefore an 
approximation is required. We are interested in fast algorithms for inferring such marginal distri- 
butions. Actually for the 2D EA model, thanks to the graph planarity, algorithms computing Z in 
a time polynomial in N exist. However we are interested in very fast (i.e. linear in N) algorithms 
that can be used also for more general model, e.g. the EA model in a field or defined on a 3D cubic 
lattice. For these more general cases a polynomial algorithm is very unlikely to exist and some 
approximations are required. 

A simple and effective mean field approximation is the one due to Bethe in which the 
marginals over the dynamic variables, like p(si), are obtained from the minimization of a variational 
free energy in a self consistent way. The Bethe approximation is exact for a model without loops 
in the interactions network, which unfortunately is far from being the usual case in physics. In 
the context of finite dimensional lattices, Kikuchi [2| derived an extension of this approximation to 
larger groups of variables, which accounts for short loops exactly, and is usually referred as Cluster 
Variational Method (CVM). 

The interest in spin glasses, with quenched random disorder, brought a new testing ground for 
both approximations. In particular Bethe approximation (exact on trees) has been the starting 
point of many useful theoretical and applied developments. It is at the basis of the cavity method, 
which allows a restatement of replica theory in probabilistic terms for finite connectivity systems 
0]. The Bethe approximation is connected to well known algorithms in computer science, namely 
Belief Propagation 0| and the sum-product algorithm [5]. A major achievement of this confluence 
between computer science and statistical mechanics, has been the conception of the Survey Propa- 
gation algorithm 0, 0] , inspired by the cavity method and the replica symmetry breaking 0, H, @] , 



that shows great performance on hard optimization problems [a. 1 71. lid. 111]. Statistical mechanics 
clarified the relation between phase transitions and easy-hard transitions in optimization problems, 
and allowed the statistical characterization of the onset of the hard phase 12|-tl4l]. as well as the 
analytical description of search algorithms based on BP [l5l . fl"tj| . 

The correctness of Bethe approximation and the related algorithms is, however, linked to the 
lack of topological correlations in the interactions (random graphs are locally tree-like), since the 
approximation is exact only on tree topologies. This is a strong limitation for physical purposes, 
since tree topologies or random graphs are not the common situation. Bethe approximation per- 
forms poorly in finite dimensional lattices, and the associated algorithm are usually non convergent 
at low temperatures. 

Recently the Cluster Variational Method (CVM) has been reformulated in a broader probabilis- 
tic framework called region-based approximations to free energy 17j and connected to a Generalized 
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Belief Propagation (GBP) algorithm to find the stationary points of the free energy. It extends 
Bethe approximation by considering correlations in larger regions, allowing, in principle, to take 
into account short loops accurately. In [17] was shown that stable fixed points of GBP message 
passing algorithm corresponds to stationary points of the approximated CVM free energy, while 
the converse is not necessarily true. Furthermore, the GBP message passing is not guaranteed to 
converge at all. Prompted by this lack of convergence, a new kind of provably convergent algo- 
rithms for minimizing the CVM approximated free energy, known as Double Loop (DL) algorithms 
has been developed, at the cost of a drastic drop off in speed. 
GBP has been applied in the last decade to inference proble ms I 2CH 22I]. consistently outper- 
forming BP. In particular, the image reconstruction problems [20l . l23l] are based on a 2D lattices 
structure, but, at variance with 2D EA model, the interactions among nearby spins (pixels) are 
ferromagnetic, and the damaged image is used as an external field. Both factors help convergence 
of GBP algorithms. An analysis of CVM approximation using GBP algorithms on single instances 
of finite dimensional disordered models of physical interest, like the EA model, has not been done 
so far. 

The Edwards Anderson model in 2D has been largely studied by other methods (see [24], [25[ and 
reference therein) suggesting that it remains paramagnetic all the way down to zero temperature, 
lacking any thermodynamic transition at any finite T, although at low T there are metastable 
states of very long lifetime, leading to very slow dynamics. Based on this fact, a paramagnetic 
version of the GBP on 2D EA model was studied recently in [26( ]. The connection of CVM with 
the replica trick and a Generalized Survey Propagation have been presented recently [27]]. However 
the implementation of the latter algorithm on finite dimensional lattices is computationally very 
demanding, and should be preceded by the study of the original CVM approximation and GBP 
algorithm. 

In this paper we study the convergence properties of GBP message passing algorithm and the 
performance of the CVM approximation on the 2D EA model. After the introduction of the 
region-based free energy in Sec. U and the message passing algorithm in terms of cavity fields, 
we compute the critical (inverse) temperature Tcvm — 0.82 (/3cvm — 1-22) of the plaquette-CVM 
approximation in Sec. IIIIl improving Bethe estimate Teethe = 1-51 (/^Bcthc — 0.66) by roughly a 
factor 2. The CVM average case temperature, however, does not clearly corresponds to the single 
instance behavior of the GBP message passing algorithm, as is shown in Sec. IIV1 At variance with 
Belief Propagation, GBP converges to spin glass solutions (below Tsg — 1.27, above /3sg — 0.79), 
and stops converging near T ~ 1.0, before the average case prediction Tcvm- In SecJV]we show that 
this convergence problem depends on the implementation details of the message passing algorithm, 
and can be improved by a simultaneous update of message. In order to do so the gauge invariance 
of the message passing equations has to be fixed. In Sec. IVII we compare the solutions and the 
performance of GBP with 3 other algorithms for the minimization of the CVM free energy: Double 
Loop [l9| , Two- Ways Message Passing [l9[ , and the Dual algorithm [26|] . In terms of the CVM free 
energy, the paramagnetic solution is in general the one to be chosen, except for a small interval in 
temperatures where the spin glass solution has a lower free energy. Our results are summarized in 
Sec. EH 



II. GENERALIZED BELIEF PROPAGATION ON EA 2D 

Given that a detailed derivation of plaquette-GBP message passing equations for the 2D Ed- 
wards Anderson model were presented in [26|, here we only summarize such derivation, skipping 
unnecessary details. 

The idea of the region-based free energy approximation fl7Ll28| is to mimic the exact (Boltzmann- 
Gibbs) distribution P(o~), by a reduced set of its marginals. A hierarchy of approximations is given 
by the size of such marginals, starting with the set of all single spins marginals Pi(si) (mean field), 
then following to all neighboring sites marginals p(si, Sj) (Bethe approximation), then to all square 
plaquettes marginals p(si, sj, Sk, si), and so on. Since the only way of knowing such marginals 
exactly is the unattainable computation of Z, the method pretends to approximate them by a set 
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of beliefs bi(si), bi{si,Sj), bp(si, Sj, Sk, sf), etc. obtained from a minimization of a region based 
free energy. 

Following the derivation done in [26], the plaquette level approximated free energy for the 2D 
EA model is given as a contribution of all Plaquettes, Links and Spins in the 2D lattice: 



PF = V V b v (a v ) log !*!fj) » Plaquettes 

EE b ^ lo § ^J^kL^ Links 



L cr L 



+EE 6 ^) lo s 



l Si 



exp(-f3E L (a L )) 

bj(sj) 
exp(-(3Ei(si)) 



(1) 



Spins 



where the symbol or = (si, . . . , S&) stands for the set of spins in region R, while -Er(ctr) = 
— S<i j>eR^ i J SiS j stands for the energy contribution in that region. The energy term Ei(si) in 
the spins contribution is only relevant when an external field acts over spins, and will be neglected 
from now on. 
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FIG. 1. Schematic representation of belief equations ([2]). Lagrange multipliers are depicted as arrows, going 
from parent regions to children regions. Italics capital letters are used to denote Plaquettes, simple capital 
letters denote Links, and lower case letters denote Spins. 



An unrestricted minimization of the free energy ([T]) in terms of its beliefs, produces incongruent 
results. Beliefs are only meaningful as an approximation to the correct marginals if they obey 
the marginalization constrains 6j(sj) = Yl Sj bi{si^Sj) and bi(si,Sj) = J2s k s t bp(si, Sj, s&, si). This 
marginalization is enforced by the introduction of Lagrange multipliers (see [l7( | for a general 
introduction, and [26] for this particular case) in the free energy expression. There is one Lagrange 
multiplier /i£_>.j(sj) for every link L and spin i € L, and a Lagrange multiplier v-p->L{sii Sj) for each 
plaquette V and link L € V . In terms of these Lagrange multipliers, the stationary condition of 
the approximated free energy is achieved with 



bi(si) = — exp ( -l3Ei(si) - y~](i L -n(si) ] , 

Ai \ LDi J 



b L ((TL) 



Z L 



exp 



V 



2 3 



VV^l{?l) ~EE ML'-+»Oi 



vdl 



iCL L'Oi 



(2) 



bvipv) = J- exp 

Zj-p 



4 1 



4 2 



-f3E v {o r ) - E E v V-+l{gl) ~ E E VL^i(si) 
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A graphical representation of these equations is given in figure [TJ Lagrange multipliers are shown 
as arrows going from parent regions, to children. Take, for one, the middle equation for the 
belief in link regions 6l(<7l) = Sj). The sum of the two Lagrange multipliers z/p^(sj, Sj) 

corresponds to the triple arrows on both sides of the link in central figure [H while the two sums 
over three messages [iL'^i( s i) corresponds to the three arrows acting over the top (J) and bottom 
(i) spins, respectively. In equations ([2]), the Zr are normalization constants. The terms Ej>(a-p) = 
E v (si,Sj,s k ,si) = -(JijSiSj + J jt kSjS k + Jk,iSkSi + Ji,iSiSi) and E L (si,Sj) = -JijSiSj are the 
corresponding energies in plaquettes and links respectively, and are represented in the diagram by 
the lines (interactions) between circles (spins), zero since no field is acting upon spins. 



u 




<D 



FIG. 2. Message passing equations J5|) and ([5]), shown schematically. Messages are depicted as arrows, 
going from parent regions to children regions. On any link Jij, represented as bold lines between spins 
(circles), a Boltzmann factor e^ Ji - jSiSj exists. Dark circles represent spins to be traced over. Messages from 
plaquettes to links fp^hisi, Sj) are represented by a triple arrow, because they can be written in terms of 
three parameters U , Ui and Uj, defining the correlation (siSj) and magnetizations (st) and (sj), respectively. 



The Lagrange multipliers can be parametrized in terms of cavity fields u and (U,u a ,Ub) as 

- HL^i(si) = /3u L ^i Si (3) 
-u v ^ L (si, sj) = P{U V ^ L SiSj + ipp-n Si + up^j sj) (4) 

In particular, the field ui^i corresponds to the cavity field in the Bethe approximation 171 ] . The 
choice of these parametrization is the reason for the use of single and triple arrows in figures [1] 
and[2j In particular, the messages going from plaquettes to links, are characterized by three fields 
(U-p^.L,u-p->i,u-p-+j), and the capital Up^L acts as an effective interaction term. 

The Lagrange multipliers are related among them by the constrains they are supposed to impose 
(see [26]). In terms of the cavity fields and using the notation in figure [2j Link-to-Spin cavity fields 
shall be related by 

UL^i = u{up-,i + U C ^i, Up^L + Uc^L + Jij, Up^j + UC-+j + UA^j + U B ^j + Ujj^j) , (5) 

where 

„, TT ^ 1 . cosh/3((7 + /i) 

Note that the usual cavity equation for fields in the Bethe approximation |3[ is recovered if all 
contributions from plaquettes V and C are set to zero. 

Similarly, by imposing the marginalization of the beliefs at Plaquettes onto their children Links, 
we find the self consistent expression for the Plaquette-to-Link cavity fields: 

Up^ L = [/(#) = — log 



4/3 ° K(l,-l)K(-l,l) 



. / ,m 1 , K(1,1)K(1,-1) 

Up^i = ~U D ^i + «;(#) = U V ^i ~ U D ^i + — log K \)K(-l -1) O 

- /„n 1 , K{1,1)K{-1,1) 

up^j = -uu->j + Uj{#) = uu^j ~ uu^j + 73 log ■ 



4/3 K(l r l)^(-l,-l) 
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where 



K(si,Sj) = ^2 ex P 



& ( (#w->a + Jjk)sjS k + (Un^R + Jki)skSi + {U V ^D + Ju)siSi + 



and the symbol # stands for all incoming fields in the right hand side of the equations. The 
functions u(u,U,h) and [U(jf), %(#)] will be used in next section for the average case 

calculation. 

For a given system of size N (number of spins) there are 2N Links and N square plaquettes, and 
therefore there are AN Plaquette-to-Link fields [U-p^L, u~p->i, u V-±j\-> an d 47V Link-to-Spins fields 
ul-h- At the stationary points of the free energy their values are related by the set of AN + AN 
equations and ([6]) . 

The set of AN + AN self-consistent equations are also called message-passing equations when 
they are used as update rules for fields in the message passing algorithm, or cavity iteration 
equations in the context of cavity calculations. The field notation is more comprehensible than the 
original Lagrange multipliers notation, and has a clear physical meaning: each plaquette is telling 
its children links that they should add an effective interaction term U p^,l to the direct interaction 
Jij, due to the fact that spins Si and Sj are also interacting through the other three links in the 
plaquette. Terms Ui act like magnetic fields upon spins, and the complete u(si, Sj)— message is 
characterized by the triplet (Uij,Ui,Uj). 



III. CRITICAL TEMPERATURE OF PLAQUETTE-CVM APPROXIMATION 

In this section we revisit the method used in [271 ] to compute the critical temperature at which 
CVM approximation develops a spin glass phase. By spin glass phase we mean a phase characterized 

by non zero local magnetizations m; = tanh f/3 u L-^i^ an d nearly zero total magnetization 

m= jr mi — (remember we work with no external field). The 2D EA model is paramagnetic 
down to zero temperature, but spin glass like solutions can appear in the CVM approximation 



due to its mean field character. We correct one of the conclusions reached in 23], where we fail 
to observe the appearance of the spin glass phase in the CVM approximation to the 2D Edwards 
Anderson model. We follow an average case approach, which is similar in spirit but different from 
the single instance stability analysis done in [2£|] for the Bethe approximation (Belief Propagation). 

The average case calculation is a mathematical technique developed in 0] to study the typical 
solutions of cavity equations in disordered systems, with a deep and fundamental connection to 
the replica trick [9(]. When applied to the plaquette-CVM approximation [23], we end up with two 
equations, in which fields (messages) are now replaced by functions of fields q(u) and Q(U,ui,U2), 
and the interactions are averaged out. As a consequence of the homogeneity of the 2D lattice 
and the averaging over local disorder Jij, all plaquettes, links, and spins in the graph are now 
equivalent, and we only need to study one of them to characterize the whole system. 

More precisely, the average case self consistent equations for the distribution q(u) is given by 

q(ui) = Ej J dq(u A ->j) dq(u B ->j) dq(uu^j) (7) 



with u(#) as defined in the right hand side of equation ([5|), and df(x) = f(x)dx 
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The corresponding self-consistent equation for Q(U,u\,u 2 ) is 

Q(U, u a , u b )q(ui - u a )q(uj - u b )du a du b = (8) 

= Ej J dq(u c ^ k ) dq(u E -,k) dqiup^) dq(u G ^i) dQ(U u ^u,uu-*j,u u ^ k ) 

dQ(U<ji->R,UK-+k,un-*i) dQ(Uv^D,uv^i,uv^i)S[U - &(#)) 5{ui - S\Uj - uj(# 

where the notation corresponds to equation ([6]). In both equations (J7J) and (|8j) the expression 
Ej = J dJP(J) . . . stands for the average over the quenched randomness. 

At high temperatures we expect fixed point equations ([5]) and ([6]) to yield a paramagnetic 
solution. Such a solution is characterized by Link to Site messages u = 0, and Plaquette to Link 
messages (U, 111,112) = (U, 0,0). If we impose this ansatz to fields, we recover the paramagnetic or 
dual algorithm of [26( for the single instance message passing, and the paramagnetic average case 
study of [27] for the average case. Let us remember that the 2D EA model is expected to have no 
thermodynamic transition at any finite temperature, and hence remain paramagnetic all the way 
down to T = 0. Following |27fl . in the average case the paramagnetic solution has the form 

q(u) = 5{u) 
Q(U, Ul , U2 ) = Q(U)5( Ul )5(u 2 ) 

The equation ([7]) is always satisfied when q(u) = 5(u) for whatever Q(U). The equation ([5]) can 
be solved self-consistently for Q{U): 

Q(U) =EjJ dQ(U u ) dQ(U n ) dQ(U v ) (9) 



<) { U — — arctanh 



tanh/3(J w + U u ) tanh/3(J^ + U n ) tanh/3(J-D + U v ) 



and the average free energy and all other relevant functions can be derived in terms of it (see [2j 

On the other hand, a general (not paramagnetic) solution of the average case equations ([7|) and 
([8]) is very difficult, since it involves the deconvolution of distributions q(u) in the left hand side 
of eq. ([8]) in order to update Q{U,u\,U2) by an iterative method. A critical temperature can be 
found, however, by an expansion in small u around the paramagnetic solution. We can focus on 
the second moments of the distributions 

\{u)u 2 du 

<Hj(U) = J J Q(U, m, U2) Ui Uj duidu2 where i,j € {1, 2} 

and check whether the paramagnetic solution (a = and aij(U) = 0) is locally stable. To do this 
we expand equations ([7|) and © to second order, and we obtain the following linearized equations: 

a = K a , a a + j dU' K a , ail (U')a u (V) + J dU' K a , ai2 (U')a 12 (U') 

aQ(U) + a u (U)=K au>a (U)a + J dU' K aiimi (U, U')a n {U') + J dU' K ailAl2 {U, U')a 12 (U') 

a 12 (U)=K ai2>a (U)a + J dV K ai2mi (U, U')a n {U') + J dU' K ai2 , ai2 (U, V')a 12 {U') 

The actual values of the K axAy come from the expansion in small u of the original equations (see 
equation 90 in 27j for an example). 
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We can not solve these equations analytically because we do not have an analytical expression of 
Q(U) for the paramagnetic solution at all temperatures. By discretizing the values of U uniformly 
in (— i7 max , U max ), i.e. U = iAU with i G [-/ ma xi4ax]i we can transform the continuous set of 
equations to a system of the form 

a = K(/3)-a (10) 
where the vector of the second moments a = (a, aii(£/), a\2{U)) have the form 

a = (a,aii(-[/ max ),aii(-C/ max + AU), . . . ,a u (U max - AU), an(C/ max ), 

ai2(-^max),ai2(-^max + AU), . . . ,ai 2 (C/ ma x ~ AC/) , a i2 (C/max)) 

K(/3) is a (2J max + l) x (2/ max + l) matrix, that stand for the discrete representation of the integrals 
in the right hand side of the linearized equations, and depends on the inverse temperature via the 
solution Q(U) of eq. ©. 

The paramagnetic solution a = always satisfy the homogeneous eq. (|10p . The stability 
criterion for the paramagnetic solution is the singularity of the Jacobian det(I — K(/3)) = 0. When 
such condition is satisfied, a non paramagnetic solution continuously arises from the paramagnetic 
one, since a flat direction appears in the free energy. 

Det[J] 




FIG. 3. Determinant of the Jacobian J = I K(/3) as a function of inverse temperature (3. The critical 
inverse temperature is /?cvm — 1-22. 

Numerically, we worked with a discretization of 2/ max + 1 = 41 points between (— C/ max = 
—3.5, [/max = 3.5). The paramagnetic solution Q(U) is found solving eq. ([9]) by an iterative 
method at every temperature, and then used to compute the elements of the K(/3) matrix. In 
figure [3] we show the determinant of the Jacobian matrix J = I — K(/3). The critical inverse 
temperature derived from this analysis is /3cvm — 1.22 for the appearance of a flat direction in the 
free e nerg y. 

In [23] /3cvm was thought to be infinite (zero temperature) because an incomplete range of 
the values of (3 was examined. The critical temperature found here is below the Bethe critical 
temperature /?Bethe — 0.66, and therefore improves the Bethe approximation by roughly a factor 
2, since the 2D EA model is likely to remain paramagnetic at all finite temperatures. At variance 
with the Bethe approximation, the single instance behavior of the message passing is not so clearly 
related to the average case critical temperature, as we show in the next section. 

IV. PERFORMANCE OF GBP ON 2D EA MODEL 

Before studying GBP message passing for the plaquette-CVM approximation, let us check what 
happens to the simpler Bethe approximation and the corresponding message passing algorithm 
known as Belief Propagation (BP) in the 2D EA model. When running BP at high temperatures 
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Probability of convergence GBP 




FIG. 4. Probability of convergence of BP and GBP on a 2D EA model, with random bimodal interactions, 
as a function of inverse temperature j3 = 1/T. The Bethe spin glass transition is expected to occur at 
/?Bcthc — 0.66 on a random graph with the same connectivity. The BP message passing algorithm on 2D 
EA model stops converging very close to that point. Above that temperature, BP equations converge to 
the paramagnetic solution, i.e. all messages are trivial u = 0. Below the Bethe temperature (nearly) the 
Bethe instability takes messages away from the paramagnetic solution, and the presence of short loops is 
thought to be responsible of the lack of convergence. On the other hand, the GBP equations converge at 
lower temperatures, but eventually stops converging as well. 

(above Teethe = V/^Bethe — 1-51) in a typical instance of the model with bimodal interactions, we 
find the paramagnetic solution (given by all fields u = 0), and therefore, the system is equivalent to 
a set of independent interacting pairs of spins, which is only correct at infinite temperature. The 
Bethe temperature Teethe (computed in average case and exact on acyclic graphs [30[]), seems to 
mark precisely the point where BP stops converging (see Fig. HJ. Indeed messages flow away from 
zero below Teethe > and convergence of the BP message passing algorithm is not achieved anymore. 
So, the Bethe approximation is disappointing when applied to single instances of the Edwards 
Anderson model: either it converge to a paramagnetic solution at high temperatures, or it does 
not converge at all below Te e the- 

The natural question arises, as to what extent GBP message passing algorithm for the plaquette- 
CVM approximation is also non convergent below its critical temperature, and whether this tem- 
perature coincides with the average case one. To check this we used GBP message passing equations 
([5]) and ([6]), with a damping factor 0.5 in the Link-to-Site fields u: 

C» = 0.5^ i + 0.5fi(#) 

We will make the distinction between two types of solutions for the GBP algorithm. The high 
temperature or paramagnetic solution is characterized by zero local magnetization of spins m; = 

Ylsi s ih( s i) = tanh Y^l u L->i\ = 0. At low temperatures, following the average case anal- 
ysis, a non paramagnetic or spin-glass solution should appear, characterized by non zero local 
magnetizations, but roughly null global magnetization. The temperature at which non zero local 
magnetizations appear will be called Tgc = V /3sg • 

Figure 0] shows that GBP is able to converge below the Bethe critical temperature, but stops 
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FIG. 5. Data points correspond to the fraction of SG solutions in a population of 100 systems of sizes 16 2 , 
32 2 , 64 2 , 128 2 , 256 2 respectively. At high temperatures (low /?) GBP message passing converge always to the 
paramagnetic solution. The average case critical inverse temperature /3cvm — 1-22 does not corresponds to 
the single instance behavior, as the spin glass solutions in GBP appear around /?sg — 0.79. The inset shows 
that all data collapsed if plotted as a function of the scaling variable L°- 9 (f3 — 0.79), where the exponent 0.9 
and the critical inverse temperature (3sg — 0.79 are obtained from best data collapse. 



converging before the CVM average case critical temperature /3cvm — 1-22. Furthermore, figure [5] 
shows that even before stop converging, GBP finds a spin glass solution in most instances. 

The inner plot of figure [5] shows a collapse of the data points for different system sizes using the 
scaling variable L°' 9 (/3 — 0.79), which gives an estimate Psg — 0.79 (the exponent 0.9 is obtained 
from the best data collapse). Since /3sg — 0.79 is well below the average case inverse critical 
temperature /3cvm — 1-22, the relevance of the latter on the behavior of GBP on single samples is 
questionable. By a similar data collapse procedure, we estimate the non-convergence temperature 
for the GBP message passing algorithm to be /3 conv ~ 0.96 (see Fig. [9|), which is again far away 
from the average case prediction /?sg- 

So, beyond the simple Bethe approximation, we found three different temperatures in the CVM 
approximation: /?sg — 0.79 < (3 conv ~ 0.96 < ficvM — 1-22 corresponding respectively to the 
appearance of spin glass solutions, to the lack of convergence on single instances, and to the 
average case prediction for the critical temperature. 

We can summarize three main differences between the properties of BP and GBP. At high 
temperatures (below /?sg — 0.79) GBP gives a quite good approximate of the marginals [261 ] . 
namely the paramagnetic solution with non trivial correlations fields U ^ 0, while BP treats the 
system as a set of independent pairs of linked spins. Furthermore, this naive approach is all that 
BP can do for us, since above /?Bethe — 0.66, it no longer converges. GBP, on the other hand, is 
not only able to converge beyond /3B e the> but it is also able to find spin glass solutions above /3sg- 
The third difference between both algorithms is that the non convergence of BP seems to occur 
exactly at the same temperature where a spin glass phase should appear (and arguably because 
of it), while the GBP convergence problems appear deep into the spin glass phase. The lack of 
convergence of GBP, however, seems to depend strongly on implementation details as we show 
next. 
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V. GAUGE INVARIANCE OF GBP EQUATIONS 

The convergence properties of the GBP message passing is sensitive to implementation details, 
e.g. the damping value in the update equations, and this is not an inherent property of the CVM 
(or region-graph) approximation. We might try, for instance, to update simultaneously all small- 
u fields pointing towards a given spin, hoping to gain some more stability in message passing 
algorithm. When trying to do this we find out that there is a freedom in the choice of these 
fields that has no effect over the fixed point solutions. This freedom (similar to the one noticed in 
[3l| ) is the result of having introduced unnecessary Lagrange multipliers to enforce marginalization 
constraints that were already indirectly enforced. 

T» 1\ T« 

/ 

• -• < • 

I i D I 



FIG. 6. Null modes of the plaquette CVM free energy in terms of fields. The small-u fields that act over a 
given spin i inside a plaquette can be shifted by an arbitrary amount S as in equation (1111) without changing 
the self consistent (message passing) equations. 

Consider, for instance, the messages shown in figureEl If the belief on a plaquette &p(sj, Sj, Sk,si) 
correctly marginalizes to the beliefs of two of its children links bi{si,Sj) and bn(si,Si), and one 
of those beliefs marginalizes to the common spin bi(si) = ^ s ■^i( s i) s j)> ft is inevitable that 
the second link D also marginalizes to the same belief on Sj, since bi(si) = ^2 Sj bL(si, Sj) = 
J2 S - St s bv(si, Sj, Sk, si) = J2 Sl t>D(si,Si). Therefore the Lagrange multiplier that was introduced 
to force this last marginalization is not needed. This redundancy is a general feature of GBP 
equations when there are more than two level of regions (Plaquette, Links, and Spins, in our case). 

The consequence of having introduced unnecessary multipliers leads to a gauge invariance on 
the fields (messages) values. Such invariance can be better understood by looking at the GBP 
equations at infinite temperature: for /3 = the non linear parts of the message passing equations 
([5]) and ([6]) disappear, but there is still a set of linear equations to be satisfied for the small-u 
messages with infinite many non trivial solutions. These solutions correspond, however, to the 
same physical paramagnetic solution, since the total field hi = X^L n -k->* an d the magnetizations 
rrii = tanh(/3/ij) are always zero. It is easy to check that once we have a solution of the message 
passing equations © and ([6]) at any temperature, we can change by an arbitrary amount 5 any 
group of 4 u-messages inside a plaquette (figure ED pointing to the same spin as 

UL-H -> U L ^i + 5 , 

uv h -+i -> u Vl ^i + 5 , (11) 

UV D ^i — > U-p D ^i - 5 , 

and still all self-consistent equations are satisfied. 

This local null mode of the standard GBP equations can be avoided by arbitrary setting to zero 
one of the four small- ii fields entering equation (|lip . We choose to fix the gauge by removing the 
right small-n field in every Plaquette-to-Link field (U, u\ e f t , it r i g ht)> as shown in figure [71 Once the 
gauge is fixed, the fields are uniquely determined, and we can try to implement the simultaneous 
updating of all small-u fields around a given spin, hopefully improving convergence. 
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FIG. 7. In the left diagram, all 8 small-u messages pointing to the central spin are highlighted with bold 
face. They are 4 Link-Site it-messages, and 4 Plaquette-Link ui e f t -messages. They have linear dependence 
among them. The right diagram shows four plaquettes around a spin, and the messages that contribute in 
a non linear way to the aforementioned 8 messages. The idea of GBP+GF is to compute the non linear 
contributions to the message passing equations, and then assign the values of the M-messages in order to 
satisfy their linear relations. 



In the left diagram of figure [7] all messages involving the central spin are represented, and in 
bold face those that act precisely upon that spin. These messages enter linearly in the message 
passing equations of each other (see equations ([5]) and ©). Therefore, the self consistent equations 
they should satisfy at the fixed points, can be written as (using the notation of figure [7j) 

ui = u a + NL\ u a = u b -u 2 + NL a 

U2 =u b + NL 2 u b = u c -u 3 + NL b , 12 . 

^3 =u c + NL 3 u c = u d -U4 + NL C \ * 

ua =Ud + NL A Ud =u a -u\ + NL d 

where the NL stand for the non linear contributions to the corresponding equation. As a con- 
sequence, the values of the 8 u-messages pointing to the central spin can be assigned precisely 
by a linear transformation for any given values of the non linear contributions. This gauge fixed 
updating method, that we will call GBP+GF, updates all u-messages around a spin simultaneously 
and in a way that they are consistent with each other via the message passing equations. 

The right diagram in figure [7] shows the messages entering the non linear parts. Taking the 8 
li-messages as zero, the non linear contributions are the right hand sides of the message passing 
equations involved. With the non linear parts computed, the system of equations (fl"2]) is solved 
for the w-variables multiplying the non linearities vector by the corresponding matrix. The 8 it- 
messages are then updated, usually with a damping factor. The update of the U correlation fields 
is done as in the original GBP method, via the equation 0, since it does not depend on the 
it-messages that are being updated. 

Figure [S] shows the probability of convergence versus inverse temperature for GBP and 
GBP+GF, and also the fraction of the solutions found that correspond to a spin glass phase. 
Let us emphasize here that GBP and GBP+GF are not different approximations, but different 
methods to find the same fixed point solution by message passing. They are expected to find 
the same solutions, and in fact they do. At high temperatures both methods converge to the 

paramagnetic solution, with all null local magnetizations m« = tanh (^(3 ux_^J = 0. The 

standard message passing update of GBP equations hardly converges above (3 conv ~ 0.96, while 
the GBP+GF method reaches lower temperatures, /? C onv-GF — 1-2, as can be seen in Fig. [9J Fur- 
thermore, the GBP+GF allows us to work in a range of temperatures where most solutions are 
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2D EA model L = 64 
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FIG. 8. Convergence probability of GBP and GBP+GF as a function of j3. The solution found by either 
iteration method is always the same (when both converge), but GBP+GF reaches lower temperatures while 
converging. The fraction of spin glass solutions found by either algorithm show that GBP+GF sees the 
same spin glass transition temperature. The fraction of spin glass solutions is always given respect to the 
amount of convergent solutions. 
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FIG. 9. Estimate of the non convergence temperature for different system sizes using the standard GBP 
(squares) and the Gauge Fixed GBP (circles) . As shown, with the gauge fixed procedure the non convergence 
extrapolated temperature is quite close to the average case prediction /?cvm — 1-22. Each data point 
corresponds to the average of the non convergence temperature over many realizations of the disorder: 10 
realizations for the 512 x 512 systems, 20 for the 256 x 256 and 100 for the others. 
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spin glass like. This proves that the non converging temperature found for GBP, /3 conv ~ 0.96, 
is not a feature of the CVM approximation, but a characteristic of the message passing method 
used, and can be outperformed by other message passing schemes, like GBP+GF. Kindly note in 
figure [9j that the non convergence inverse temperature of GBP+GF /3 coi1v _gf — 1-2 is quite close to 
the average case prediction for the critical temperature /3c vm — 1-22. Whether this is accidental 
or not is still unclear. Since the average case instability should describe the breakdown of the 
paramagnetic phase, and the lack of convergence in single instances occurs while already in a non 
paramagnetic phase, it seems far fetched assuming that both critical behaviors are related. 



The disagreement between the average case critical temperature /?cvm and the one observed 
in the single instance /?sg> can be due to a number of reasons. First, the average case calculation 
assumes that cavity fields are uncorrelated. But, in our case, messages participating in the cavity 
iteration are very close to each other in the lattice, and thus correlated. Furthermore, GBP does 
not have the equivalent of a Bethe lattice for BP, i.e. a model in which the correlation between 
cavity messages is close to zero by construction. The second reason for a failure of the average 
case prediction is that the transition we observe in single instances might be due to the almost 
inevitable appearance of ferromagnetic domains in large systems (Griffith instability). The third, 
and the most obvious reason, is that the gauge invariance was not accounted in the average case 
calculation. 

Reproducing the method of Sec. Illll to obtain an average case prediction of the critical temper- 
ature for the Gauge Fixed GBP is not straightforward. The reason is that Link-to-Spins messages 
u, should fulfill two different equations: their own original equation ([5]), and the implicit equation 
derived from the fact that the gauge is fixed and one of the fields in the Plaquette-to-Link message 
(U, u, u) is set to zero. 



FIG. 10. Left: The set of four messages that we compute jointly by a population dynamic. Right: the 
population dynamic step consists in taking four quadruplets at random from the population (those in black) , 
and computing a new quadruplet (the one in gray inside the plaquette) using randomly selected interactions 
Jij on the plaquette. 

However, a different average case calculation is possible. We can represent the messages flowing 
in the lattice by a population of quadruplets (ui^ijUp-^i, Up^.i r ,UL r ^r), where one of the original 
messages is absent because the gauge has been fixed (see left panel in Fig. fTOj) . Given any four 
of these quadruplets of messages around a plaquette, we can compute, using the message passing 
equations, the new messages inside the plaquette (see right panel in Fig. HO]) . The new population 
dynamics consists in picking four of these quadruplets out of the population at random, then 
computing the new quadruplet (using also random interactions in the plaquette) and finally put 



A. Gauge fixed average case stability 
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FIG. 11. Edwards Anderson order parameter, see eq. ([T3l) . obtained using a population of N = 10 3 messages, 
and running the population dynamic step 10 3 x N times. In agreement with the single instance behavior, the 
transition between paramagnetic (qEA — 0) and non paramagnetic (spin glass) phases is found at j3 ~ 0.78. 

it back in the population. After several steps, the population stabilizes either to a paramagnetic 
solution (where all u = and only U ^ 0), either to a non paramagnetic one (where also «^0). 

In Fig. [TT] we show the Edwards Anderson order parameter qEA = obtained at 

different temperatures using this population dynamics average case method. We find that qEA 
becomes larger than zero at /?cvm-gf — 0.78, which is quite close to the inverse temperature 
/3sg — 0.79 where single instances develop non-zero local magnetizations and a spin glass phase. 
The correspondence between this average case result and the single instance behaviour is very 
enlightening: indeed the average case computation does not take into account correlations among 
quadruplets of messages and it is not sensible to Griffith's singularities. So, the most simple 
explanation for the GBP-GF behaviour on single samples of the 2D EA model is that quadruplets 
of messages arriving on any given plaquette are mostly uncorrelated and that at /3sg a true spin 
glass instability takes place (which is an artifact of the mean- field like approximation). Please 
consider that under the Bethe approximation the SG instability happens at /?Bethe — 0.66, while 
the CVM approximation improves the estimate of the SG critical boundary to /?sg — 0.79 (on 
single instances) and to /?cvm-gf — 0.78 (on the average case). 



VI. SAME APPROXIMATION, FOUR ALGORITHMS 

It can be proved [17[ that stable fixed points of the message passing equations correspond to 
stationary points of the region graph approximated free energy (or CVM free energy). The converse 
is not necessarily true, and some of the stationary points of the free energy, might not be stable 
under the message passing heuristic. As we have seen, the message passing might not even converge 
at all. For a given free energy approximation (eq. ([T|) in our case), there are other algorithms to 
search for stationary points, including other types of message passing and provably convergent 
algorithms. In this section we study two of these algorithms and show that they do find the same 
spin glass like transition at f3 m , but have a different behavior at lower temperatures. 

The one presented so far is the so called Parent-to-Child (PTC) message passing algorithm, 
in which Lagrange multipliers are introduced to force marginalization of bigger (parent) regions 
onto their children. Other choices of Lagrange multipliers are possible [13], leading to the so 
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FIG. 12. Free energy of the solutions found by Double Loop algorithm, HAK and the GBP PTC algorithm 
relative to the free energy of the paramagnetic solution (Dual approximation), in a typical system in which 
GBP PTC finds a spin glass solution. At high temperatures both algorithm find the same paramagnetic 
solution. Interestingly, there is a small range of temperatures where the spin glass solution found by GBP is 
actually the one that minimizes the free energy. But at even lower temperatures the paramagnetic solution 
becomes again the correct one. While Double Loop and HAK switch back to the paramagnetic solution 
(even if at a wrong T), the GBP PTC get stuck in the spin glass solution (and for this reason, it eventually 
stops converging). 

called Child-to-Parent and Two- Ways algorithms. Next we test the following four algorithms for 
minimizing the plaquette-CVM free energy in typical instances of 2D EA: 

• Double-Loop algorithm of Heskes et. al. Is a provably convergent algorithm that 
guarantees a step by step minimization of the free energy functional. It consist of two loops, 
the inner of which is a Two- Ways message passing algorithm that we will call HAK. We use 
the implementation in LibDai public library [32] |. 

• HAK message passing algorithm. Is a Two- Ways message passing algorithm [lj|. When it 
converges, it is usually faster than Double-Loop. 

• GBP Parent-to-Child is the message passing algorithm we have presented so far in this paper, 
and for which the simultaneous updating of cavity fields was introduced to help convergence. 
Nevertheless the following results were obtained using standard GBP PTC. 

• Dual algorithm of [26]. Is the same GBP PTC setting all small fields u = 0, and doing only 
message passing in terms of correlation fields U (first equation in eq. ©). 

For the last three algorithms we use our own implementation in terms of cavity fields u and 
(U, u a ,Uf)). The dual algorithm forces the solution of GBP to remain paramagnetic since all u = 0. 
This paramagnetic ansatz is specially suited for the 2D EA model since it is expected to be 
paramagnetic at any finite temperature (in the thermodynamical limit). 

As shown in the previous section, the GBP PTC message passing equations finds a paramagnetic 
solution in the 2D EA model at high temperatures, while below TgQ = 1//?sg — 1-27 it finds a 
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FIG. 13. Convergence time in seconds for the Double Loop algorithm (full points) and standard message 
passing algorithms (empty points) for the plaquette-GBP approximation in two different realizations of a 
16 2 Edwards Anderson system. Message passing algorithms are typically faster, but not always convergent. 
The first cusp is related to the appearance of the spin glass solution, while the second cusp in the Double 
Loop algorithm is related to the switching back to the paramagnetic solution (see Fig. [12]) . 



spin glass like solution. By spin glass like we mean that the total field hi = Y^*L u L^i and the 
magnetization rni = tanh(/3/tj) are non zero and change from spin to spin. The order parameter 

qEA = ]v ^ m * ( 13 ^ 

i 

is used to locate this phase. The critical temperature Tsg, where qEA becomes larger than zero, 
seems to be independent of message passing details, like damping or the use of gauge fixing for 
simultaneous updates of fields. 

In figure [12] we show the free energy and the qEA parameter of the solutions found by Double 
Loop, HAK and GBP PTC for two typical realizations of an N = 16 x 16 EA system with bimodal 
interactions. The free energy of the dual approximation is subtracted to highlight the differences 
with respect to the paramagnetic solution. The figure shows that HAK and Double Loop do find 
the same spin glass solution that GBP PTC finds when going down in temperature. This solution is 
actually lower in free energy when it appears, but at even lower temperatures becomes subdominant 
compared to the paramagnetic one. The GBP PTC keeps finding the spin glass solutions while 
Double Loop and HAK switch back to the paramagnetic one. This is an interesting feature of 
Double Loop and in particular of HAK which is a fast message passing algorithm. By returning to 
the dual (paramagnetic) solution, HAK is also ensuring its convergence at low temperature [261 ]. 
while GBP PTC get lost in the irrelevant (and physically wrong) spin glass solution, and eventually 
stops converging. 

However note that DL and HAK may stop finding the SG solution when this solution is still 
the one with lower free energy. Moreover the lack of convergence of GBP can be used as a warning 
that something wrong is happening with the CVM approximation, something that is impossible to 
understand by looking at the behavior of provably convergent algorithms. 

In figure [T51 we compare the running times of Double Loop (LibDai [12]), HAK and GBP PTC 
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(our implementation) for the two systems of figure [T2l As expected, Double Loop is much more 
slowly than the message passing heuristics of HAK and GBP (please notice the log scale in the 
time axis). The peaks in running times correspond to the transition points from paramagnetic 
to spin glass solution. Double Loop and HAK have two peaks, the second corresponding to the 
transition back to paramagnetic solution, while the GBP PTC has only the first peak. 



VII. SUMMARY AND CONCLUSIONS 

We studied the properties of the Generalized Belief Propagation algorithm derived from a 
Cluster Variational Method approximation to the free energy of the Edwards Anderson model in 
2D at the level of plaquettes. We compared the results obtained by Parent-to-Child GBP with the 
ones obtained by the Dual (paramagnetic) algorithm [26[ and by HAK Two- Ways algorithm [3] 
and Double-Loop provably convergent algorithm [l9l |. 

We found that the plaquette-CVM approximation (using Parent-to-Child GBP) is far richer 
than the Bethe (BP) approximation in 2D EA model. BP converges only at high temperatures 
(above Teethe = V/^Bethe = 1-51), and in such case it treats the system as a set of independent pairs 
of linked spins. GBP on the other hand, makes a better prediction on the paramagnetic behavior 
of the model at high T, since it implements a message passing of correlations fields flowing from 
plaquettes to links in the graph. Furthermore with GBP the paramagnetic phase is extended to 
temperatures below Teethe = 1-51 until Tgc = 1//3sg — 1-27 where spin glass solutions appear 
in the single instance implementation of the message passing algorithm. In contrast to Bethe 
approximation, GBP is able to find spin glass solutions, and the standard message passing stops 
converging near T conv ~ 1. 

The average case calculation of the stability of the paramagnetic solution in the CVM approxi- 
mation predicted that non paramagnetic (spin glass) solutions should appear at lower temperatures 
Tcvm = 1//?CVM — 0.82. This average case result does not coincide with the single instance be- 
havior of the standard GBP, since it fails to mark both the point where GBP start finding spin 
glass solutions Tsg and the point where GBP stops converging T conv . 

However, the non convergence of GBP is not a feature of the CVM approximation, and is 
susceptible of changes from one implementation of the message passing to another. We showed 
that by fixing a hidden gauge invariance in the message passing equation, a simultaneous update of 
all cavity fields pointing to a single spin in the lattice improves the convergence of the algorithm, 
without changing drastically its speed. Using the gauge fixed GBP, the non convergence inverse 
temperature is moved to T conv _GF — 1.2, quite close to the average case prediction Tcvm (whether 
this is only a coincidence is still not clear). Most importantly the average case computation 
(population dynamics) with the gauge fixed identifies the same SG critical temperature Tcvm-gf — 
1.28 measured on single samples (where Tbg — 1.27). 

Finally we compared the fixed point solutions found by the GBP message passing with those 
found by the provably convergent Double-Loop algorithm and the message passing heuristic of the 
Two- Ways algorithm of [l{|. All the algorithms find the same paramagnetic solutions at high T, 
while below TgG they find a spin glass solution, in the sense that local magnetizations are non 
zero, while the global magnetization is null. Decreasing the temperature Double-Loop and HAK 
switch back from the spin glass to the paramagnetic solution, at the cost of a factor 10 2 — 10 3 and 
10 — 10 2 respectively in running time, compared to GBP. Furthermore, the paramagnetic solution 
can always be found fast by the Dual algorithm of [26] , making these two algorithms (Double-Loop 
and HAK) unnecessarily slow. 

Although the thermodynamics of the 2D EA model is paramagnetic, at low temperatures, the 
correlation length grows until eventually surpassing L/2 and therefore being effectively infinite for 
any finite size 2D system. In such a situation the non paramagnetic solutions obtained by GBP 
can account for long range correlations, and presumably gives better estimates for the correlations 
among spins than the paramagnetic solution obtained by HAK and Double Loop. 

Establishing the previous claim requires a detailed study of the quality of CVM approximation at 
low temperatures (in the non paramagnetic range) and its connections to the statics and dynamics 
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of 2D Edwards Anderson model, which is already under study. Application of CVM and GBP 
message passing to Edwards Anderson model in 3D is also appealing, since this model does have 
a spin glass behavior at low temperature. 



[1] H. A. Bethe, Proc. R. Soc. A. 150, 552 (1935). 

[2] R. Kikuchi, Phys. Rev. 81, 988 (1951). 

[3] M. Mezard and G. Parisi, Eur. Phys. J. B 20, 217 (2001). 

[4] J. Pearl, Probabilistic Reasoning in Intelligent Systems (2nd ed.) (Morgan Kaufmann, San Francisco, 
CA, 1988). 

[5] F. R. Kschischang, B. J. Frey, and H. Loeliger, IEEE Trans. Inform. Theory 47, 498 (1998). 

[6] M. Mezard and R. Zecchina, Phys. Rev. E 66, 056126 (2002). 

[7] M. Mezard, G. Parisi, and R. Zecchina, Science 297, 812 (2002). 

[8] M. Mezard and G. Parisi, J. Stat. Phys. 111,1 (2003). 

[9] G. Parisi, M. Mezard, and M. A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 
1987). 

10] R. Mulet, A. Pagnani, M. Weigt, and R. Zecchina, Phys. Rev. Lett. 89, 268701 (2002). 

11] A. Braunstein, R. Mulet, A. Pagnani, M. Weigt, and R. Zecchina, Phys. Rev. E 68, 036702 (2003). 

12] D. Achlioptas, A. Naor, and Y. Peres, Nature 435, 759 (2005). 

13] F. K rzakala, A. M ontanar i, F. Ricci-Tersenghi, G. Semerjian, and L. Zdeborova, 



Proc. Nat. Acad. Sri. 104. lO.TLs (20(17) 



[14] A. Montanari, F. Ricci-Tersenghi, and G. Semerjian, [J. Stat Mech , P 04004 (2008). 
[15] A. Montanari, F. Ricci-Tersenghi, and G. Semerjian, in Proceedings of the 45th Annual Allerton 

Conference on Communication, Control, and Computing (2007) pp. 352-359. 
[16] F. Ricci-Tersenghi and G. Semerjian, | J. Stat. Mech. , P09001 (2009) 
[17] J. Yedidia, W. T. Freeman, and Y. Weiss, IEEE Trans. Inform. Theory 51, 2282 (2005). 
[18] Y. S.-K. Eye and A. L. Yuille, Neural Comput. 14, 2002 (2001). 
[19] T. Heskes, C. A. Albers, and H. J. Kappen, UAI-03 , 313 (2003). 

[20] K. Tanaka, J. Inoue, and D. M. Titterington, in XIII Proceedings of the 2003 IEEE Signal Processing 

Society Workshop (17-19 September, 2003 (IEEE Com puter Society Pre ss, 2003) pp. 329-338. 
[21] C. A. Albers, T. Heskes, and H. J. Kappen, [Genetics 177, 1101 (2007)] 
[22] H. J. Kappen, in Modeling Bio-medical signals (World Scientific, 2002) pp. 3-16. 
[23] K. Tanaka and T. Morita, |Phys. Lett. A 203, 122 (1995)[ 

[24] T. Jorg, J. Lukic, E. Marinari, and O. Martin, Phys. Rev. Lett. 96, 237205 (2006). 

[25] C. K. Thomas and A. A. Middleton, |Phys. Rev. E 80, 046708 (200gJ _ 

[26] A. Lage-Castellanos, R. Mulet, F. Ricci-Tersenghi, and T. Rizzo, [Phys. Rev. E , to appear (2011) 
[27] T. Rizzo, A. Lage-Castellanos, R. Mulet, and F. Ricci-Tersenghi, |j. Stat. Phys. 139, 375 (2010)| 

[28] A. Pelizzola, J. Phys. A 38, R3 09 (2005). 

[29] J. M. Mooij and H. J. Kappen, [XStat. Mech P11012 (2005)] 

[30] The Bethe temperature Tsethe is the one at which a non trivial spin glass solution appears for a random 

regular Bethe lattice with connectivity K = 4. The Bethe lattice looks locally as a tree. 
[31] M. Weigt, R. A. Whitea, H. Szurmantc, J. A. Hochc, and T. Hwa, Proc. Nat. Acad. Sci. 36, 11096 
(2003). 



[32] J. M. Mooij, J. Mach. Learn. Res. 11, 2169 (2010) 



